Skip to content

End spans that a plugin leaks after its interceptor returns or BeforeInvoke fails, so tracing continues - #266

Merged
wu-sheng merged 2 commits into
apache:mainfrom
mrproliu:reclaim-leaked-spans
Nov 12, 2026
Merged

wu-sheng merged 2 commits into
apache:mainfrom
mrproliu:reclaim-leaked-spans

Conversation

@mrproliu

@mrproliu mrproliu commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #265.

Motivation

Since #265, a request that is ignored, or that runs while the agent is disconnected, stores a noop span as the active span. If a plugin leaks that span, every later span on the same goroutine is also a noop. With the default config, that goroutine then stops tracing for good. A plugin can leak a span in two ways:

  • BeforeInvoke stores the span with SetContext, but AfterInvoke never ends it.
  • BeforeInvoke creates a span, then returns an error or panics before calling SetContext.

Leaked real spans have a similar effect: they stay on the stack and become the parent of unrelated spans.

Changes

  • The interception owns its stored span. SetContext binds the stored span to the current interception, and the generated after-code marks that interception as exited. The next time a span is created, or ActiveSpanMark is called, the core ends every active span whose interception has already exited. It logs a warning for each one and counts it in sw_go_possible_leaked_context_counter. One reclaim pass ends at most 64 spans. A continued snapshot is never ended.
  • Failed BeforeInvoke is rolled back. The before-template records the depth of the active span stack before calling BeforeInvoke. If BeforeInvoke returns an error or panics without calling SetContext, the spans it created are ended. A context that the hook continued with ContinueContext is kept.
  • A reused entry span stays active. When a nested interceptor reuses the entry span and stores it with SetContext, ending the nested call no longer pops the span. It stays active until the outer request ends. Callers that only create and end the span keep their current behavior, for example kratos middleware, consumers that pull messages inside a request, and the toolkit.
  • grpc-v184: RecvMsg keeps its span active on purpose until the trailer is read. It now stores the span in a holder, so the span is not reclaimed when RecvMsg returns.
  • Rewriter: the rewriter now recognizes all 18 Go builtins. Before this change, close in the copied operator code was renamed to a package function, which broke instrumented builds.
  • Docs: added a "Span lifecycle" section to the development guide, and three CHANGES.md entries.

Tests

  • New unit tests:
    • core: reclaiming leaked spans, rollback, snapshot and continued contexts, and reused entry spans;
    • operator and tracing bridge;
    • rendering of the before/after templates;
    • rewriter handling of builtins.
  • make test and make test-race pass. golangci-lint v2.15.0 reports no issues.
  • All plugin scenarios were run locally, and the results match main:
    • kratosv2, gin, http, grpc, gorm, so11y, cross-goroutine, and the rest pass with unchanged expected data.
    • pulsar passes when it runs alone.
    • segmentio-kafka has a startup race between the app and Kafka. main shows the same race.
    • pprof fails on main too. It expects http.params, but the scenario doesn't enable parameter collection.

@mrproliu mrproliu added this to the 0.8.0 milestone Nov 9, 2026
@mrproliu
mrproliu requested a balanced review from Copilot November 9, 2026 13:27
@mrproliu mrproliu added the bug Something isn't working label Nov 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

7 open issues
What changed in this PR

This PR addresses leaked spans in Go agent interceptors so tracing doesn’t get “stuck” (especially with noop spans) after buggy plugins or failed BeforeInvoke hooks.

Changes:

  • Add span ownership + orphan reclamation/rollback mechanisms to prevent leaked spans from affecting later requests on the same goroutine.
  • Update code generation templates to mark interceptor exit and to rollback spans created by failed/panicking BeforeInvoke.
  • Fix rewriter builtin handling and add extensive unit tests + documentation for span lifecycle rules.
File Description
tools/​go-agent/​instrument/​plugins/​templates/​method_intercept_before.tmpl Marks/rolls back active span around BeforeInvoke and marks invocation exited on skip path.
tools/​go-agent/​instrument/​plugins/​templates/​method_intercept_after.tmpl Ensures invocations are marked exited even when AfterInvoke panics.
tools/​go-agent/​instrument/​plugins/​rewrite/​context.go Prevents renaming of all Go predeclared builtin functions during rewrite.
tools/​go-agent/​instrument/​plugins/​rewrite/​context_test.go Adds test coverage ensuring builtins are not renamed in rewritten agent-core copies.
tools/​go-agent/​instrument/​plugins/​enhance_method_test.go Validates generated before/after template code ordering (mark, rollback, and markExited).
plugins/​grpc-v184/​client_recvmsg_interceptor.go Wraps RecvMsg span in a holder so it can intentionally remain active until trailer read.
plugins/​core/​tracing/​bridge.go Adds span-owner binding hook (BindOwner) at the tracing bridge wrapper layer.
plugins/​core/​tracing/​bridge_test.go Tests that SpanWrapper.BindOwner forwards ownership when supported by the wrapped span.
plugins/​core/​tracing.go Ensures orphan spans are ended before creating a new span under an active context.
plugins/​core/​span_owner.go Introduces orphan detection, reclaim, and rollback logic; adds owner binding for spans/noops.
plugins/​core/​span_owner_test.go Adds extensive tests for rollback/reclaim behavior across ignored/noop/continued/reused spans.
plugins/​core/​span_noop.go Tracks noop span ownership + continued-snapshot marker for reclaim/rollback rules.
plugins/​core/​span_default.go Tracks reuse owners and ownership-stored status to keep reused entry spans active appropriately.
plugins/​core/​operator/​invocation.go Binds span ownership to invocations via SetContext; adds markExited and rollback helpers.
plugins/​core/​operator/​invocation_test.go Tests span ownership binding and rollback helper behavior.
plugins/​core/​operator/​intercepter_test_base.go Adds a helper to simulate generated interception order for tests.
docs/​en/​development-and-contribution/​development-guide.md Documents interceptor span lifecycle rules and how reclaim/rollback works.
CHANGES.md Release notes for span leak reclamation, reused-entry behavior, and builtin rewrite fix.

Review effort: Lite

Comment thread plugins/core/span_owner.go Resolved
Comment on lines +139 to +143
if t.Log != nil {
t.Log.Warnf("ending the span %q that its plugin did not end", name)
}
span.End()
}
Comment thread plugins/core/span_owner_test.go Resolved
@mrproliu mrproliu changed the title End spans that a plugin leaks after its interceptor returns or Before Invoke fails, so tracing continues End spans that a plugin leaks after its interceptor returns or BeforeInvoke fails, so tracing continues Nov 9, 2026
@wu-sheng
wu-sheng merged commit 5d7175f into apache:main Nov 12, 2026
82 of 86 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants